{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "<center>\n",
    "<img src=\"https://github.com/oussou-dev/mlcourse.ai/blob/jupyter_french/img/ods_stickers.jpg?raw=true\" />\n",
    "<br>\n",
    "    \n",
    "<div style=\"font-weight:700;font-size:25px\"> [mlcourse.ai](https://mlcourse.ai) - Open Machine Learning Course </div>\n",
    "\n",
    "<br>\n",
    "Auteur: [Yury Kashnitsky](https://yorko.github.io). Traduit et édité par [Christina Butsko](https://www.linkedin.com/in/christinabutsko/), [Nerses Bagiyan](https://www.linkedin.com/in/nersesbagiyan/), [Yulia Klimushina](https://www.linkedin.com/in/yuliya-klimushina-7168a9139), [Yuanyuan Pao](https://www.linkedin.com/in/yuanyuanpao/) et [Ousmane Cissé](https://github.com/oussou-dev). Ce matériel est soumis aux termes et conditions de la licence [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). L'utilisation gratuite est autorisée à des fins non commerciales."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "# <center>Topic 4. Classification et régression linéaires\n",
    "## <center> Partie 4. Où la régression logistique est bonne et où elle ne l'est pas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "### Analyse des critiques de films IMDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "Maintenant pour un peu de pratique! Nous voulons résoudre le problème de la classification binaire des critiques de films IMDB. Nous avons un training set avec des critiques marquées, 12500 critiques marquées comme bonnes, 12500 comme mauvaises. Ici, il n’est pas facile de se lancer immédiatement dans l’apprentissage automatique car nous n’avons pas la matrice $X$; nous devons la préparer. Nous allons utiliser une approche simple: modèle de \"sac de mots\" (bag of words model). Les caractéristiques de la critique seront représentées par des indicateurs de la présence de chaque mot de l'ensemble du corpus dans cette critique. Le corpus est l'ensemble de toutes les critiques d'utilisateurs. L'idée est illustrée par une photo\n",
    "\n",
    "<img src=\"https://github.com/oussou-dev/mlcourse.ai/blob/jupyter_french/img/bag_of_words.png?raw=true\" width=80% />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_files\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Pour commencer, nous téléchargeons automatiquement le jeu de données depuis [ici](http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz) et le désarchivons avec le reste des jeux de données. dans le dossier de données. L'ensemble de données est brièvement décrit [ici](http://ai.stanford.edu/~amaas/data/sentiment/). Il y a 12,5k de bonnes et de mauvaises critiques dans les données de test et de training set.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tarfile\n",
    "from io import BytesIO\n",
    "\n",
    "import requests\n",
    "\n",
    "url = \"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\"\n",
    "\n",
    "\n",
    "def load_imdb_dataset(extract_path=\"../../data\", overwrite=False):\n",
    "    # check if existed already\n",
    "    if (\n",
    "        os.path.isfile(os.path.join(extract_path, \"aclImdb\", \"README\"))\n",
    "        and not overwrite\n",
    "    ):\n",
    "        print(\"IMDB dataset is already in place.\")\n",
    "        return\n",
    "\n",
    "    print(\"Downloading the dataset from:  \", url)\n",
    "    response = requests.get(url)\n",
    "\n",
    "    tar = tarfile.open(mode=\"r:gz\", fileobj=BytesIO(response.content))\n",
    "\n",
    "    data = tar.extractall(extract_path)\n",
    "\n",
    "\n",
    "load_imdb_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change if you have it in alternative location\n",
    "PATH_TO_IMDB = \"../../data/aclImdb\"\n",
    "\n",
    "reviews_train = load_files(\n",
    "    os.path.join(PATH_TO_IMDB, \"train\"), categories=[\"pos\", \"neg\"]\n",
    ")\n",
    "text_train, y_train = reviews_train.data, reviews_train.target\n",
    "\n",
    "reviews_test = load_files(os.path.join(PATH_TO_IMDB, \"test\"), categories=[\"pos\", \"neg\"])\n",
    "text_test, y_test = reviews_test.data, reviews_test.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Alternatively, load data from previously pickled objects.\n",
    "# import pickle\n",
    "# with open('../../data/imdb_text_train.pkl', 'rb') as f:\n",
    "#     text_train = pickle.load(f)\n",
    "# with open('../../data/imdb_text_test.pkl', 'rb') as f:\n",
    "#     text_test = pickle.load(f)\n",
    "# with open('../../data/imdb_target_train.pkl', 'rb') as f:\n",
    "#     y_train = pickle.load(f)\n",
    "# with open('../../data/imdb_target_test.pkl', 'rb') as f:\n",
    "#     y_test = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of documents in training data: %d\" % len(text_train))\n",
    "print(np.bincount(y_train))\n",
    "print(\"Number of documents in test data: %d\" % len(text_test))\n",
    "print(np.bincount(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Voici quelques exemples d’avis.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(text_train[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[1]  # bad review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_train[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[2]  # good review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pickle\n",
    "# with open('../../data/imdb_text_train.pkl', 'wb') as f:\n",
    "#     pickle.dump(text_train, f)\n",
    "# with open('../../data/imdb_text_test.pkl', 'wb') as f:\n",
    "#     pickle.dump(text_test, f)\n",
    "# with open('../../data/imdb_target_train.pkl', 'wb') as f:\n",
    "#     pickle.dump(y_train, f)\n",
    "# with open('../../data/imdb_target_test.pkl', 'wb') as f:\n",
    "#     pickle.dump(y_test, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "## Un simple décompte de mots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Tout d'abord, nous allons créer un dictionnaire de tous les mots en utilisant CountVectorizer**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer()\n",
    "cv.fit(text_train)\n",
    "\n",
    "len(cv.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Si vous regardez les exemples de \"mots\" (appelons-les jetons), vous constaterez que nous avons omis bon nombre des étapes importantes du traitement de texte (le traitement de texte automatique peut lui-même constituer une série d'articles complètement distinct). .**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cv.get_feature_names()[:50])\n",
    "print(cv.get_feature_names()[50000:50050])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Deuxièmement, nous encodons les phrases des textes du training set avec les index des mots entrants. Nous utiliserons le format \"clairsemé\".**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = cv.transform(text_train)\n",
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Voyons comment notre transformation a fonctionné**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(text_train[19726])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train[19726].nonzero()[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train[19726].nonzero()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Troisièmement, nous appliquerons les mêmes opérations au jeu de test**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = cv.transform(text_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**La prochaine étape consiste à entraîner la régression logistique.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "logit = LogisticRegression(solver=\"lbfgs\", n_jobs=-1, random_state=7)\n",
    "logit.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's look at accuracy on the both the training and the test sets.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "round(logit.score(X_train, y_train), 3), round(logit.score(X_test, y_test), 3),"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Les coefficients du modèle peuvent être magnifiquement affichés.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualize_coefficients(classifier, feature_names, n_top_features=25):\n",
    "    # get coefficients with large absolute values\n",
    "    coef = classifier.coef_.ravel()\n",
    "    positive_coefficients = np.argsort(coef)[-n_top_features:]\n",
    "    negative_coefficients = np.argsort(coef)[:n_top_features]\n",
    "    interesting_coefficients = np.hstack([negative_coefficients, positive_coefficients])\n",
    "    # plot them\n",
    "    plt.figure(figsize=(15, 5))\n",
    "    colors = [\"red\" if c < 0 else \"blue\" for c in coef[interesting_coefficients]]\n",
    "    plt.bar(np.arange(2 * n_top_features), coef[interesting_coefficients], color=colors)\n",
    "    feature_names = np.array(feature_names)\n",
    "    plt.xticks(\n",
    "        np.arange(1, 1 + 2 * n_top_features),\n",
    "        feature_names[interesting_coefficients],\n",
    "        rotation=60,\n",
    "        ha=\"right\",\n",
    "    );"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_grid_scores(grid, param_name):\n",
    "    plt.plot(\n",
    "        grid.param_grid[param_name],\n",
    "        grid.cv_results_[\"mean_train_score\"],\n",
    "        color=\"green\",\n",
    "        label=\"train\",\n",
    "    )\n",
    "    plt.plot(\n",
    "        grid.param_grid[param_name],\n",
    "        grid.cv_results_[\"mean_test_score\"],\n",
    "        color=\"red\",\n",
    "        label=\"test\",\n",
    "    )\n",
    "    plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_coefficients(logit, cv.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Pour améliorer notre modèle, nous pouvons optimiser le coefficient de régularisation pour la «régression logistique». Nous utiliserons `sklearn.pipeline` car` CountVectorizer` ne devrait être appliqué qu'aux données d'apprentissage (afin de ne pas \"jeter un coup d'œil\" dans le jeu de test et de ne pas y compter la fréquence des mots). Dans ce cas, `pipeline` détermine la séquence d'actions appropriée: applique` CountVectorizer`, puis entraîne `Régression logistique`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "text_pipe_logit = make_pipeline(\n",
    "    CountVectorizer(),\n",
    "    # for some reason n_jobs > 1 won't work\n",
    "    # with GridSearchCV's n_jobs > 1\n",
    "    LogisticRegression(solver=\"lbfgs\", n_jobs=1, random_state=7),\n",
    ")\n",
    "\n",
    "text_pipe_logit.fit(text_train, y_train)\n",
    "print(text_pipe_logit.score(text_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid_logit = {\"logisticregression__C\": np.logspace(-5, 0, 6)}\n",
    "grid_logit = GridSearchCV(\n",
    "    text_pipe_logit, param_grid_logit, return_train_score=True, cv=3, n_jobs=-1\n",
    ")\n",
    "\n",
    "grid_logit.fit(text_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Imprimons les meilleurs $C$ et cv-score en utilisant cet hyperparamètre:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_logit.best_params_, grid_logit.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_grid_scores(grid_logit, \"logisticregression__C\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the validation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_logit.score(text_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "**Maintenant, faisons la même chose avec la forêt aléatoire. Nous voyons qu'avec la régression logistique, nous obtenons une meilleure précision avec moins d'effort.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forest = RandomForestClassifier(n_estimators=200, n_jobs=-1, random_state=17)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "forest.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "round(forest.score(X_test, y_test), 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "### XOR-Problem\n",
    "Considérons maintenant un exemple où les modèles linéaires sont pires.\n",
    "\n",
    "Les méthodes de classification linéaire définissent encore une surface de séparation très simple - un hyperplan. L'exemple de jouet le plus célèbre où les classes ne peuvent pas être divisées par un hyperplan (ou une ligne) sans erreur est \"le problème XOR\".\n",
    "\n",
    "XOR est le \"OU exclusif\", une fonction booléenne avec la table de vérité suivante:\n",
    "\n",
    "\n",
    "\n",
    "<img src='https://github.com/oussou-dev/mlcourse.ai/blob/jupyter_french/img/XOR_table.gif?raw=true'>\n",
    "\n",
    "XOR est le nom donné à un simple problème de classification binaire dans lequel les classes sont présentées sous forme de nuages ​​de points intersectants étendus en diagonale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating dataset\n",
    "rng = np.random.RandomState(0)\n",
    "X = rng.randn(200, 2)\n",
    "y = np.logical_xor(X[:, 0] > 0, X[:, 1] > 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X[:, 0], X[:, 1], s=30, c=y, cmap=plt.cm.Paired);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "De toute évidence, on ne peut pas tracer une seule ligne droite pour séparer une classe d'une autre sans erreurs. Par conséquent, la régression logistique fonctionne mal avec cette tâche."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_boundary(clf, X, y, plot_title):\n",
    "    xx, yy = np.meshgrid(np.linspace(-3, 3, 50), np.linspace(-3, 3, 50))\n",
    "    clf.fit(X, y)\n",
    "    # plot the decision function for each datapoint on the grid\n",
    "    Z = clf.predict_proba(np.vstack((xx.ravel(), yy.ravel())).T)[:, 1]\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    image = plt.imshow(\n",
    "        Z,\n",
    "        interpolation=\"nearest\",\n",
    "        extent=(xx.min(), xx.max(), yy.min(), yy.max()),\n",
    "        aspect=\"auto\",\n",
    "        origin=\"lower\",\n",
    "        cmap=plt.cm.PuOr_r,\n",
    "    )\n",
    "    contours = plt.contour(xx, yy, Z, levels=[0], linewidths=2, linetypes=\"--\")\n",
    "    plt.scatter(X[:, 0], X[:, 1], s=30, c=y, cmap=plt.cm.Paired)\n",
    "    plt.xticks(())\n",
    "    plt.yticks(())\n",
    "    plt.xlabel(r\"$x_1$\")\n",
    "    plt.ylabel(r\"$x_2$\")\n",
    "    plt.axis([-3, 3, -3, 3])\n",
    "    plt.colorbar(image)\n",
    "    plt.title(plot_title, fontsize=12);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(\n",
    "    LogisticRegression(solver=\"lbfgs\"), X, y, \"Logistic Regression, XOR problem\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "Mais si l’on devait entrer des caractéristiques polynomiales (ici, jusqu’à 2 degrés), le problème est résolu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logit_pipe = Pipeline(\n",
    "    [\n",
    "        (\"poly\", PolynomialFeatures(degree=2)),\n",
    "        (\"logit\", LogisticRegression(solver=\"lbfgs\")),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(logit_pipe, X, y, \"Logistic Regression + quadratic features. XOR problem\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "Ici, la régression logistique a toujours produit un hyperplan mais dans un espace à 6 dimensions $1, x_1, x_2, x_1^2, x_1x_2$ et $x_2^2$. Lorsque nous projetons vers l'espace de caractéristiques d'origine, $x_1, x_2$, la limite est non linéaire.\n",
    "\n",
    "En pratique, les caractéristiques polynomiales aident, mais il est inefficace en calcul de les construire explicitement. SVM à noyau fonctionne beaucoup plus rapidement. Dans cette approche, seule la distance entre les objets (définie par la fonction du noyau) dans un espace de grande dimension est calculée et il n'est pas nécessaire de produire un nombre combinatoire important de caractéristiques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "fr"
   },
   "source": [
    "### Ressources utiles\n",
    "- Main course [site](https://mlcourse.ai), [course repo](https://github.com/Yorko/mlcourse.ai), and YouTube [channel](https://www.youtube.com/watch?v=QKTuw4PNOsU&list=PLVlY_7IJCMJeRfZ68eVfEcu-UcN9BbwiX)\n",
    "- Medium [\"story\"](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-4-linear-classification-and-regression-44a41b9b5220) based on this notebook\n",
    "- Course materials as a [Kaggle Dataset](https://www.kaggle.com/kashnitsky/mlcourse)\n",
    "- If you read Russian: an [article](https://habrahabr.ru/company/ods/blog/323890/) on Habrahabr with ~ the same material. And a [lecture](https://youtu.be/oTXGQ-_oqvI) on YouTube\n",
    "- A nice and concise overview of linear models is given in the book [“Deep Learning”](http://www.deeplearningbook.org) (I. Goodfellow, Y. Bengio, and A. Courville).\n",
    "- Linear models are covered practically in every ML book. We recommend “Pattern Recognition and Machine Learning” (C. Bishop) and “Machine Learning: A Probabilistic Perspective” (K. Murphy).\n",
    "- If you prefer a thorough overview of linear model from a statistician’s viewpoint, then look at “The elements of statistical learning” (T. Hastie, R. Tibshirani, and J. Friedman).\n",
    "- The book “Machine Learning in Action” (P. Harrington) will walk you through implementations of classic ML algorithms in pure Python.\n",
    "- [Scikit-learn](http://scikit-learn.org/stable/documentation.html) library. These guys work hard on writing really clear documentation.\n",
    "- Scipy 2017 [scikit-learn tutorial](https://github.com/amueller/scipy-2017-sklearn) by Alex Gramfort and Andreas Mueller.\n",
    "- One more [ML course](https://github.com/diefimov/MTH594_MachineLearning) with very good materials.\n",
    "- [Implementations](https://github.com/rushter/MLAlgorithms) of many ML algorithms. Search for linear regression and logistic regression."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  },
  "nbTranslate": {
   "displayLangs": [
    "*"
   ],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "en",
   "targetLang": "fr",
   "useGoogleTranslate": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
